PVFS2 over Quadrics: Design, Implementation and Performance Evaluation

Parallel I/O needs to keep pace with the everincreasing computing power of high performance computers for the real world applications to perform well. Exploiting high-end interconnect technologies such as Quadrics to reduce the network access cost and scale the aggregated bandwidth is one of the ways to increase the performance of storage systems. In this paper, we explore the challenges of supporting parallel file system with modern features of Quadrics, including user-level communication and RDMA operations. We then design and implement a Quadrics-capable version of a parallel file system (PVFS2) by overcoming Quadrics static communication model and providing an efficient transport layer over Quadrics, which includes an optimized noncontiguous IO support using multiple zero-copy RDMA operations with a chained event. Experiment results indicates that, with Quadrics user-level protocols and RDMA operations, the performance of PVFS2 is significantly improved in terms of both data transfer and management operations. With four IO server nodes, our implementation improves PVFS2 aggregated read bandwidth by up to 140% compared to PVFS2 over TCP/IP. Moreover, it delivers significant performance improvement in terms of IO access to application benchmarks such as mpi-tile-io [24] and BT-IO [26]. To the best of our knowledge, this is the first work in the literature to report the design of a high performance parallel file system over Quadrics user-level communication protocols.